Determining Malignancy of Pulmonary Nodules using Deep Learning

ABSTRACT

Systems and method are described for determining a malignancy of a nodule. A medical image of a nodule of a patient is received. A patch surrounding the nodule is identified in the medical image. A malignancy of the nodule in the patch is predicted using a trained deep image-to-image network.

TECHNICAL FIELD

The present invention relates generally to determining malignancy ofpulmonary nodules, and more particularly to determining malignancy ofpulmonary nodules by analyzing medical images using machine learningmodels.

BACKGROUND

The current standard for screening a patient for lung cancer is computedtomography (CT) imaging. If pulmonary nodules are found in the CTimaging of the patient, a biopsy may be performed to retrieve portionsof the nodules to determine their malignancy by histopathologicalexamination. The decision to perform a biopsy is based on simplefeatures of the CT imaging, such as the number of nodules, the size ofthe nodules, the shape of the nodules, and the growth of the nodules.However, such simple features of the CT imaging constitute a smallamount of the total information available in the CT imaging. Asignificant amount of the information available in the CT imagingremains unused in determining whether to perform a biopsy.

Pulmonary biopsies are an expensive procedure to perform. For manypatients who undergo biopsies, their nodules are found to be benign.Reducing the number of patients who undergo such unnecessary biopsieswould result in significant medical cost savings, while also reducingpatient exposure to unnecessary medical procedures.

BRIEF SUMMARY OF THE INVENTION

In accordance with one or more embodiments, systems and methods fordetermining a malignancy of a nodule are provided. A medical image of anodule of a patient is received and a patch surrounding the nodule isdefined in the medical image. A malignancy of the nodule in the patch ispredicted using a trained deep image-to-image network. In oneembodiment, the trained deep image-to-image network comprises a deepreasoner network.

In accordance with one embodiment, the deep image-to-image network istrained using training images depicting particular nodes and results ofa histopathological examination of the particular nodules. The deepimage-to-image network may additionally or alternatively be trained(e.g., where the results of the histopathological examination of theparticular nodules are insufficient to train the deep image-to-imagenetwork) using additional training images depicting additional nodulesand results of a radiologist examination of the additional nodules.

In accordance with one embodiment, the malignancy of the nodule ispredicted by determining a score indicating the malignancy of the noduleor by classifying the nodule as malignant or not malignant. Based on thepredicted malignancy of the nodule, another medical image of the noduleof the patient may be received. For example, the other medical image maybe received in response to the predicted malignancy of the nodule (e.g.,a score indicating a malignancy of the nodule) satisfying one or morethresholds. The other medical image may be a more detailed medical imageof the nodule than the initial medical image. Another patch surroundingthe nodule may be defined in the other medical image and the malignancyof the nodule in the other patch may be predicted using the trained deepimage-to-image network.

In accordance with one or more embodiments, the trained deepimage-to-image network may comprise an encoder and a decoder. The patchis input into the encoder and the encoder converts the patch to a lowlevel representation. The decoder predicts the malignancy of the nodulein the patch from the low level representation.

These and other advantages of the invention will be apparent to those ofordinary skill in the art by reference to the following detaileddescription and the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an illustrative system for determining a malignancy ofnodules, in accordance with one or more embodiments;

FIG. 2 shows a high-level workflow for predicting a malignancy of apulmonary nodule, in accordance with one or more embodiments;

FIG. 3 shows a method for predicting a malignancy of a nodule of apatient, in accordance with one or more embodiments;

FIG. 4 shows an exemplary medical image of a lung of a patient having anodule, in accordance with one or more embodiments;

FIG. 5 shows network architecture of a deep image-to-image network forpredicting a malignancy of a nodule of a patient, in accordance with oneor more embodiments;

FIG. 6 shows a high-level network architecture for predicting amalignancy of a nodule of a patient, in accordance with one or moreembodiments;

FIG. 7 shows a workflow for training and applying a machine learningmodel for predicting a malignancy of a nodule of a patient, inaccordance with one or more embodiments;

FIG. 8 shows a workflow for generating training data for training amachine learning model for predicting a malignancy of a nodule of apatient, in accordance with one or more embodiments; and

FIG. 9 shows a high-level block diagram of a computer.

DETAILED DESCRIPTION

The present invention generally relates to methods and systems fordetermining malignancy of pulmonary nodules using deep learning.Embodiments of the present invention are described herein to give avisual understanding of such methods and systems. A digital image isoften composed of digital representations of one or more objects (orshapes). The digital representation of an object is often describedherein in terms of identifying and manipulating the objects. Suchmanipulations are virtual manipulations accomplished in the memory orother circuitry/hardware of a computer system. Accordingly, is to beunderstood that embodiments of the present invention may be performed bya computer system using data stored within the computer system.

FIG. 1 shows a system 100 configured to determine or predict amalignancy of nodules, in accordance with one or more embodiments.System 100 includes workstation 102, which may be used for assisting aclinician (e.g., a doctor, a medical professional, or any other user) inperforming a medical evaluation on a patient 106 (or any other subject).Workstation 102 may be implemented using any suitable computing device,such as, e.g., computer 902 of FIG. 9.

In one embodiment, workstation 102 may assist the clinician in screeningpatient 106 for lung cancer. Accordingly, workstation 102 may receivemedical images of patient 106 from one or more medical imaging systems104. Medical imaging system 104 may be of any modality, such as, e.g., atwo-dimensional (2D) or three-dimensional (3D) computed tomography (CT),x-ray, magnetic resonance imaging (MRI), ultrasound (US), single-photonemission computed tomography (SPECT), positron emission tomography(PET), or any other suitable modality or combination of modalities. Inanother embodiment, workstation 102 may receive the images by loadingpreviously stored images of the patient acquired using medical imagingsystem 104.

Embodiments of the present invention provide for the analysis of medicalimages using machine learning models to determine or predict amalignancy of pulmonary nodules. Advantageously, such machine learningmodels determine a malignancy of pulmonary nodules using features of themedical images that cannot be practically or possibly used in humananalysis under conventional clinical practice. Accordingly, embodimentsof the present invention use additional information from the medicalimages to exclude a subset of patients who would have unnecessarilyundergone biopsies under conventional clinical practices, therebyreducing medical costs and patient exposure to unnecessary medicalprocedures.

It should be understood that while the embodiments discussed herein maybe discussed with respect to analyzing medical images to determinemalignancy of pulmonary nodules of a patient, the present invention isnot so limited. Embodiments of the present invention may be applied foranalyzing any type of image for any measure of interest.

FIG. 2 shows a high-level workflow 200 for predicting a malignancy of apulmonary nodule, in accordance with one or more embodiments. Inworkflow 200, a 3D CT medical image 202 of the lungs of a patient isanalyzed to define a patch 206 encompassing a pulmonary nodule 208, asshown in exploded view 204 of medical image 204. Patch 206 encompassingpulmonary nodule 208 is input into artificial intelligence (A.I.) system210, which outputs a prediction on the malignancy of pulmonary nodule208. The prediction may be a classification of pulmonary nodule 208 asbeing malignant 214 or benign (i.e., not malignant) 212. A medicaldecision may be made based on the prediction. For example, if pulmonarynodule 208 is predicted to be malignant 214, the patient may undergo abiopsy of pulmonary nodule 208 for histopathological examination toconfirm its malignancy.

FIG. 3 shows a method 300 for predicting a malignancy of a nodule of apatient, in accordance with one or more embodiments. Method 300 will bediscussed with respect to system 100 of FIG. 1. In one embodiment, thesteps of method 300 are performed by workstation 102 of FIG. 1.

At step 302, a medical image of a nodule of a patient is received. Themedical image may be directly received from a medical imaging system,such as, e.g., medical imaging system 104 of FIG. 1. Alternatively, themedical image may be received by loading a previously acquired medicalimage from a storage or memory of a computer system or receiving amedical image that has been transmitted from a remote computer system.The medical image may be of any suitable modality, but is preferably athree-dimensional (3D) CT image. In one embodiment, the medical image isan image of the lungs of the patient depicting one or more pulmonarynodules. However, it should be understood that the medical image may beof any region of interest of the patient depicting any subject ofinterest.

At step 304, a patch surrounding the nodule is defined in the medicalimage of the patient. In one embodiment, the patch is a spatially local3D patch surrounding the nodule in the medical image. The patch may bedefined in the medical image using any suitable approach. In oneembodiment, a deep reinforcement based algorithm is applied to identifythe nodule in the medical image and define a patch surrounding thenodule. Accordingly, agents are trained to efficiently navigate themedical image to identify the nodule and define a patch surrounding thenodule. In another embodiment, 3D anisotropic hybrid networks may beutilized to leverage the full spatial resolution of the medical image toidentify the nodule and define the patch surrounding the nodule. Inanother embodiment, a user (e.g., a clinician) may manually identify thenodule and define the patch surrounding the nodule. A patch surroundinga nodule is shown in FIG. 4.

At step 306, a malignancy of the nodule in the patch is predicted usinga trained deep image-to-image network. In one embodiment, the deepimage-to-image network is a deep reasoner network trained with a densenetwork and multi-task learning to predict a malignancy of the nodule.The deep image-to-image network is described in further detail belowwith respect to FIGS. 5-7.

The prediction of the malignancy of the nodule may be in any suitableform. In one embodiment, the malignancy of the nodule is predicted byclassifying the nodule as being, e.g., malignant or not malignant (i.e.,benign) or benign or not benign (i.e., malignant). In anotherembodiment, the malignancy of the nodule is predicted by determining amalignancy score indicating a malignancy of the nodule. For example, themalignancy score may be a score between zero and one, where a lowerscore (e.g., zero) indicates a greater confidence that the nodule is notmalignant (i.e., benign) while a higher score indicates a greaterconfidence that the identified nodule is malignant.

At step 308, a medical decision for the nodule is made based on thepredicted malignancy of the nodule. The medical decision may be anysuitable medical decision. In one example, the medical decision may bewhether or not to perform a biopsy for histopathological examination onthe nodule of the patient. In one embodiment, the predicted malignancymay be used to confirm an analysis of a radiologist of the medical imagein determining whether to perform a biopsy.

In one embodiment, where the predicted malignancy is the binaryclassification of malignant or not malignant, the medical decision is toperform the biopsy for histopathological examination on the nodule wherethe predicted malignancy is malignant, and to not perform the biopsy(and take no further action) on the nodule where the predictedmalignancy is not malignant.

In another embodiment, where the predicted malignancy is a malignancyscore, the medical decision is made based on the malignancy score withrespect to one or more thresholds. For example, if the malignancy scoreis below a first threshold of, e.g., 0.2, the medical decision is to notperform the biopsy (and take no further action) on the nodule. If themalignancy score is between the first threshold and a second thresholdof, e.g., 0.5, the medical decision is to receive or acquire a moredetailed medical image (e.g., a positron emission tomography scan) ofthe nodule for further analysis. Accordingly, steps 302-308 may berepeated for the more detailed medical image. If the malignancy score isabove the second threshold, the medical decision is to perform a biopsyfor histopathological examination on the nodule.

FIG. 4 shows a medical image 400 of a lung of a patient, in accordancewith one or more embodiments. Medical image 400 depicts pulmonary nodule404. As shown in FIG. 4, patch 402 surrounding nodule 404 is identified.Patch 402 may be a spatially local 3D patch surrounding nodule 404.Patch 402 may be defined in medical image 400 using any suitableapproach, such as, e.g., deep reinforcement based algorithms or 3Danisotropic hybrid networks. In one embodiment, patch 402 is the patchidentified at step 304 of FIG. 3.

FIG. 5 illustrates a network architecture 500 of a deep image-to-imagenetwork, in accordance with one or more embodiments. In one embodiment,the deep image-to-image network is a deep reasoner network with a densenetwork. The deep image-to-image network is trained with multi-tasklearning to generate an output image J as a reconstruction of an inputimage I, and to predict a malignancy of a nodule in input image I. Inone embodiment, the deep image-to-image network is trained and appliedas described with respect to FIG. 7.

Network architecture 500 of the deep image-to-image network comprises aseries of layers 502 of an encoding network (or encoder) F and a seriesof layers 504 of a decoding network (or decoder) G. Encoder F receivesinput image I. In one embodiment, input image I is the patch identifiedat step 304 of FIG. 3 or patch 402 of FIG. 4. Layers 502 of encoder Fcode or convert the input image I to a low level representation, whosesize is substantially less than the size of the input image I. Forexample, encoder F may condense information of input image I to a smallsubset of values representing its lowest level information. Layers 504of decoder G are branched into layers 504-A for automatically generatingan output image J as a reconstruction of an input image I, and layers504-B for automatically predicting a malignancy of a nodule in inputimage I. Layers 504-A of decoder G will decode the low levelrepresentation from encoder F into an output image J, which is areconstruction of input image I. This can be expressed as J=G(F(I)).Layers 504-B of decoder G will decode the low level representation fromencoder F to predict a malignancy of a nodule in input image I. All theintermediate information generated in the encoder F is shared with thedecoder G so that no information is lost in the encoding process.

The trained network architecture 500 of the deep image-to-image networkmay be applied during an inference stage to predict a malignancy of anodule at step 306 of FIG. 3. Accordingly, layers 504-B of decoder Gwill decode the code from layers 502 of encoder F to predict themalignancy of the nodule, while layers 504-A of decoder G will not beused. Parameters of the deep image-to-image network may be adjusted to,e.g., weigh the predictions towards finding that the nodules aremalignant.

FIG. 6 shows a high-level network architecture 600 for predictingmalignancy of a nodule, in accordance with one or more embodiments. Oneor more patches 602 surrounding nodules identified from medical imagesare input in a convolutional neural network (CNN) 604, whose output isinput into CNN 606. In one embodiment, the one or more patches 602 mayinclude the patch identified at step 304 of FIG. 3 or patch 402 of FIG.4. CNNs 604 and 606 have a 2×2 filter and a stride of 2. The output ofCNN 606 is input into a deep reasoner network with a dense network 608trained to output a malignancy prediction 610 for the nodules.Malignancy prediction 610 may include, for example, classifying thenodules (e.g., malignant or not malignant), determining a malignancyscore, etc. In one embodiment, the deep reasoner network with a densenetwork 608 is represented by network architecture 500 of FIG. 5.

FIG. 7 shows a workflow 700 for training and applying a machine learningmodel, in accordance with one or more embodiments. Steps 702-706 show anoffline or training stage for training a machine learning model. Steps708-712 show an online or inference stage for applying the trainedmachine learning model on newly received medical images. In oneembodiment, step 306 of FIG. 3 is performed by performing the steps ofthe inference state (steps 708-712). The inference stage (steps 708-712)can be repeated for each newly received medical image(s). In oneembodiment, the steps of workflow 700 may performed to train and applythe network architecture 500 of the deep image-to-image network in FIG.5.

At step 702, during a training stage, training images including nodulesare received. The training images are medical images acquired using amedical imaging modality corresponding to the modality of the inputimage to be analyzed during the inference stage (at step 706). Forexample, the modality may be computed tomography (CT), magneticresonance (MR), DynaCT, ultrasound, x-ray, positron emission tomography(PET), etc. In one embodiment, the training images can be received byloading a number of previously stored medical training images from adatabase of medical images.

The training images may be annotated with ground truths indicatingwhether or not the nodules are malignant. In one embodiment, groundtruths for a respective training image are determined by performing abiopsy on the nodule shown in the respective training image andperforming histopathological examination of the biopsied nodule. Inanother embodiment, the ground truths may be determined based on aradiologist report of the biopsied nodule. While ground truthsdetermined based on a radiologist report may not be as accurate asground truths determined based on a histopathological examination, insome embodiments, ground truths determined based on a radiologist reportmay be used to supplement the ground truths determined based on ahistopathological examination, e.g., where the ground truths determinedbased on a histopathological examination are insufficient to train themachine learning model. Generating training images annotated with groundtruths is described in further detail below with respect to FIG. 8.

At step 704, patches surrounding the nodules in the training images aredefined. The patches surrounding the nodules may be defined in thetraining images using any suitable approach, such as, e.g., the methodsdescribed above with respect to step 304 of FIG. 3. For example, thepatches surrounding the nodules may be defined in the training imagesusing a deep reinforcement based algorithm or 3D anisotropic hybridnetworks.

In one embodiment, the training images received at step 702 are thepatches surrounding the nodules. In this embodiment, step 704 may beskipped and the method may proceed to step 706 using the patches as thetraining images.

At step 706, a machine learning model is trained to predict a malignancyof the nodules in the patches. In one embodiment, the machine learningmodel is a deep image-to-image network, such as, e.g., a deep reasonernetwork with a dense network. In this embodiment, the deepimage-to-image network is also trained to generate output images asreconstructions of the training images, as shown in network architecture500 of FIG. 5. However, the deep image-to-image network to generateoutput images will not be used in the inference stage.

At step 708, during an inference stage, an input medical image of apatient is received. The input medical image comprises a patchsurrounding a nodule. In one embodiment, the input medical image is thepatch surrounding the nodule identified at step 304 of FIG. 3 or patch402 of FIG. 4.

At step 710, a malignancy of the nodule in the input medical image ispredicted using the trained machine learning model. The predictedmalignancy may be a classification of the nodule (e.g., malignant or notmalignant, benign or not benign, etc.), a malignancy score indicating amalignancy of the nodule, or any other suitable prediction.

At step 712, the predicted malignancy of the nodule is output. In oneembodiment, the predicted malignancy of the nodule is output byreturning the predicted malignancy to step 306 of FIG. 3. In someembodiments, the predicted malignancy of the nodule can be output bydisplaying the predicted malignancy on a display device of a computersystem, storing the predicted malignancy on a memory or storage of acomputer system, or by transmitting the predicted malignancy to a remotecomputer system.

It should be understood that once the machine learning model is trainedduring the training stage, the steps 708-712 of the inference stage canbe repeated for each newly received input medical image(s). For example,blocks 708-712 can be repeated for each patch surrounding a nodule.

FIG. 8 shows a workflow 800 for generating training data for training amachine learning model for predicting a malignancy of a nodule of apatient, in accordance with one or more embodiments. In one embodiment,the training data generated by performing workflow 800 may be thetraining images received at step 702 of the training stage of FIG. 7.

At step 802, a first examination (Exam 1) is performed to acquire CTmedical images having pulmonary nodules for a population of patients. Atstep 804, a radiologist analyzes the CT medical images to identify thepulmonary nodules and generate a radiologist medical report. At step806, a second examination (Exam 2) is performed by performing a biopsyon the pulmonary nodules. At step 808, a histopathological examinationis performed on the biopsied pulmonary nodules to determine whether thepulmonary nodules are malignant. At step 810, the CT medical imagesacquired at step 802 are associated with the results of thehistopathological examination determined at step 808 as ground truthsfor training an A.I system (e.g., a deep image-to-image network) topredict a malignancy of the pulmonary nodules.

In some embodiments, results of the radiologist analysis determined atstep 804 may alternatively or additionally be used as the ground truths.While the results of the radiologist analysis determined at step 804 maynot be as accurate as the results of the histopathological examinationdetermined at step 808, in some embodiments, the results of theradiologist analysis can be used to supplement the results of thehistopathological examination where, for example, there is aninsufficient amount of results of the histopathological examination fortraining the A.I. system.

Systems, apparatuses, and methods described herein may be implementedusing digital circuitry, or using one or more computers using well-knowncomputer processors, memory units, storage devices, computer software,and other components. Typically, a computer includes a processor forexecuting instructions and one or more memories for storing instructionsand data. A computer may also include, or be coupled to, one or moremass storage devices, such as one or more magnetic disks, internal harddisks and removable disks, magneto-optical disks, optical disks, etc.

Systems, apparatus, and methods described herein may be implementedusing computers operating in a client-server relationship. Typically, insuch a system, the client computers are located remotely from the servercomputer and interact via a network. The client-server relationship maybe defined and controlled by computer programs running on the respectiveclient and server computers.

Systems, apparatus, and methods described herein may be implementedwithin a network-based cloud computing system. In such a network-basedcloud computing system, a server or another processor that is connectedto a network communicates with one or more client computers via anetwork. A client computer may communicate with the server via a networkbrowser application residing and operating on the client computer, forexample. A client computer may store data on the server and access thedata via the network. A client computer may transmit requests for data,or requests for online services, to the server via the network. Theserver may perform requested services and provide data to the clientcomputer(s). The server may also transmit data adapted to cause a clientcomputer to perform a specified function, e.g., to perform acalculation, to display specified data on a screen, etc. For example,the server may transmit a request adapted to cause a client computer toperform one or more of the steps or functions of the methods andworkflows described herein, including one or more of the steps orfunctions of FIGS. 2-3 and 7-8. Certain steps or functions of themethods and workflows described herein, including one or more of thesteps or functions of FIGS. 2-3 and 7-8, may be performed by a server orby another processor in a network-based cloud-computing system. Certainsteps or functions of the methods and workflows described herein,including one or more of the steps of FIGS. 2-3 and 7-8, may beperformed by a client computer in a network-based cloud computingsystem. The steps or functions of the methods and workflows describedherein, including one or more of the steps of FIGS. 2-3 and 7-8, may beperformed by a server and/or by a client computer in a network-basedcloud computing system, in any combination.

Systems, apparatus, and methods described herein may be implementedusing a computer program product tangibly embodied in an informationcarrier, e.g., in a non-transitory machine-readable storage device, forexecution by a programmable processor; and the method and workflow stepsdescribed herein, including one or more of the steps or functions ofFIGS. 2-3 and 7-8, may be implemented using one or more computerprograms that are executable by such a processor. A computer program isa set of computer program instructions that can be used, directly orindirectly, in a computer to perform a certain activity or bring about acertain result. A computer program can be written in any form ofprogramming language, including compiled or interpreted languages, andit can be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment.

A high-level block diagram of an example computer 902 that may be usedto implement systems, apparatus, and methods described herein isdepicted in FIG. 9. Computer 902 includes a processor 904 operativelycoupled to a data storage device 912 and a memory 910. Processor 904controls the overall operation of computer 902 by executing computerprogram instructions that define such operations. The computer programinstructions may be stored in data storage device 912, or other computerreadable medium, and loaded into memory 910 when execution of thecomputer program instructions is desired. Thus, the method and workflowsteps or functions of FIGS. 2-3 and 7-8 can be defined by the computerprogram instructions stored in memory 910 and/or data storage device 912and controlled by processor 904 executing the computer programinstructions. For example, the computer program instructions can beimplemented as computer executable code programmed by one skilled in theart to perform the method and workflow steps or functions of FIGS. 2-3and 7-8. Accordingly, by executing the computer program instructions,the processor 904 executes the method and workflow steps or functions ofFIGS. 2-3 and 7-8. Computer 904 may also include one or more networkinterfaces 906 for communicating with other devices via a network.Computer 902 may also include one or more input/output devices 908 thatenable user interaction with computer 902 (e.g., display, keyboard,mouse, speakers, buttons, etc.).

Processor 904 may include both general and special purposemicroprocessors, and may be the sole processor or one of multipleprocessors of computer 902. Processor 904 may include one or morecentral processing units (CPUs), for example. Processor 904, datastorage device 912, and/or memory 910 may include, be supplemented by,or incorporated in, one or more application-specific integrated circuits(ASICs) and/or one or more field programmable gate arrays (FPGAs).

Data storage device 912 and memory 910 each include a tangiblenon-transitory computer readable storage medium. Data storage device912, and memory 910, may each include high-speed random access memory,such as dynamic random access memory (DRAM), static random access memory(SRAM), double data rate synchronous dynamic random access memory (DDRRAM), or other random access solid state memory devices, and may includenon-volatile memory, such as one or more magnetic disk storage devicessuch as internal hard disks and removable disks, magneto-optical diskstorage devices, optical disk storage devices, flash memory devices,semiconductor memory devices, such as erasable programmable read-onlymemory (EPROM), electrically erasable programmable read-only memory(EEPROM), compact disc read-only memory (CD-ROM), digital versatile discread-only memory (DVD-ROM) disks, or other non-volatile solid statestorage devices.

Input/output devices 908 may include peripherals, such as a printer,scanner, display screen, etc. For example, input/output devices 908 mayinclude a display device such as a cathode ray tube (CRT) or liquidcrystal display (LCD) monitor for displaying information to the user, akeyboard, and a pointing device such as a mouse or a trackball by whichthe user can provide input to computer 902.

Any or all of the systems and apparatus discussed herein, includingelements of workstation 102 of FIG. 1, may be implemented using one ormore computers such as computer 902.

One skilled in the art will recognize that an implementation of anactual computer or computer system may have other structures and maycontain other components as well, and that FIG. 9 is a high levelrepresentation of some of the components of such a computer forillustrative purposes.

The foregoing Detailed Description is to be understood as being in everyrespect illustrative and exemplary, but not restrictive, and the scopeof the invention disclosed herein is not to be determined from theDetailed Description, but rather from the claims as interpretedaccording to the full breadth permitted by the patent laws. It is to beunderstood that the embodiments shown and described herein are onlyillustrative of the principles of the present invention and that variousmodifications may be implemented by those skilled in the art withoutdeparting from the scope and spirit of the invention. Those skilled inthe art could implement various other feature combinations withoutdeparting from the scope and spirit of the invention.

1. A method for determining a malignancy of a nodule, comprising:receiving a medical image of a nodule of a patient; defining a patchsurrounding the nodule in the medical image; and predicting a malignancyof the nodule in the patch using a trained deep image-to-image network.2. The method of claim 1, further comprising: training the trained deepimage-to-image network using training images depicting particularnodules and results of a histopathological examination of the particularnodules.
 3. The method of claim 2, wherein training the trained deepimage-to-image network using training images depicting particularnodules and results of a histopathological examination of the particularnodules further comprises: training the trained deep image-to-imagenetwork using additional training images depicting additional nodulesand results of a radiologist examination of the additional nodules. 4.The method of claim 1, wherein predicting a malignancy of the nodule inthe patch using a trained deep image-to-image network comprises:determining a score indicating the malignancy of the nodule.
 5. Themethod of claim 1, wherein predicting a malignancy of the nodule in thepatch using a trained deep image-to-image network comprises: classifyingthe nodule as malignant or not malignant.
 6. The method of claim 1,further comprising: receiving another medical image of the nodule of thepatient based on the predicted malignancy of the nodule; defininganother patch surrounding the nodule in the other medical image; andpredicting another malignancy of the nodule in the other patch using thetrained deep image-to-image network.
 7. The method of claim 6, whereinreceiving another medical image of the nodule of the patient based onthe predicted malignancy of the nodule comprises: receiving the othermedical image of the nodule of the patient based on the predictedmalignancy of the nodule with respect to one or more thresholds, thepredicted malignancy of the nodule comprising a score indicating amalignancy of the nodule.
 8. The method of claim 1, wherein predicting amalignancy of the nodule in the patch using a trained deepimage-to-image network comprises: inputting the patch into an encoder ofthe trained deep image-to-image network, the encoder converting thepatch to a low level representation; predicting the malignancy of thenodule in the patch from the low level representation by a decoder ofthe trained deep image-to-image network.
 9. The method of claim 1,wherein the trained deep image-to-image network comprises a trained deepreasoner network.
 10. An apparatus for determining a malignancy of anodule, comprising: means for receiving a medical image of a nodule of apatient; means for defining a patch surrounding the nodule in themedical image; and means for predicting a malignancy of the nodule inthe patch using a trained deep image-to-image network.
 11. The apparatusof claim 10, further comprising: means for training the trained deepimage-to-image network using training images depicting particularnodules and results of a histopathological examination of the particularnodules.
 12. The apparatus of claim 11, wherein the means for trainingthe trained deep image-to-image network using training images depictingparticular nodules and results of a histopathological examination of theparticular nodules further comprises: means for training the traineddeep image-to-image network using additional training images depictingadditional nodules and results of a radiologist examination of theadditional nodules.
 13. The apparatus of claim 10, wherein the means forpredicting a malignancy of the nodule in the patch using a trained deepimage-to-image network comprises: means for determining a scoreindicating the malignancy of the nodule.
 14. The apparatus of claim 10,wherein the means for predicting a malignancy of the nodule in the patchusing a trained deep image-to-image network comprises: means forclassifying the nodule as malignant or not malignant.
 15. Anon-transitory computer readable medium storing computer programinstructions for determining a malignancy of a nodule, the computerprogram instructions when executed by a processor cause the processor toperform operations comprising: receiving a medical image of a nodule ofa patient; defining a patch surrounding the nodule in the medical image;and predicting a malignancy of the nodule in the patch using a traineddeep image-to-image network.
 16. The non-transitory computer readablemedium of claim 15, the operations further comprising: training thetrained deep image-to-image network using training images depictingparticular nodules and results of a histopathological examination of theparticular nodules.
 17. The non-transitory computer readable medium ofclaim 15, the operations further comprising: receiving another medicalimage of the nodule of the patient based on the predicted malignancy ofthe nodule; defining another patch surrounding the nodule in the othermedical image; and predicting another malignancy of the nodule in theother patch using the trained deep image-to-image network.
 18. Thenon-transitory computer readable medium of claim 17, wherein receivinganother medical image of the nodule of the patient based on thepredicted malignancy of the nodule comprises: receiving the othermedical image of the nodule of the patient based on the predictedmalignancy of the nodule with respect to one or more thresholds, thepredicted malignancy of the nodule comprising a score indicating amalignancy of the nodule
 19. The non-transitory computer readable mediumof claim 15, wherein predicting a malignancy of the nodule in the patchusing a trained deep image-to-image network comprises: inputting thepatch into an encoder of the trained deep image-to-image network, theencoder converting the patch to a low level representation; predictingthe malignancy of the nodule in the patch from the low levelrepresentation by a decoder of the trained deep image-to-image network.20. The non-transitory computer readable medium of claim 15, wherein thetrained deep image-to-image network comprises a trained deep reasonernetwork.